18.05 Lecture 23 
April 6, 2005 



Estimation Theory: 

If only 2 outcomes: Bernoulli distribution describes your experiment. 
If calculating wrong numbers: Poisson distribution describes experiment. 
May know the type of distribution, but not the parameters involved. 

A sample (i.i.d.) X\, ...,X n has distribution P from the family of distributions: 

{F e : e e e} 

P = Pe , 6*o is unknown 

Estimation Theory - take data and estimate the parameter. 
It is often obvious based on the relation to the problem itself. 

Example: B(p), sample: 0011010111 
p = E(X) ^:r = 6/10 = 0.6 

Example: E(a), ae"" 1 , x > Q,E(X) = I /a. 

Once again, parameter is connected to the expected value. 

1/a = E(X) <— x, a m 1/x - estimate of alpha. 

Bayes Estimators: - used when intuitive model can be used in describing the data. 




* 1 

G 

X u ...,X n ~V eo ,9 G 6 

Prior Distribution - describes the distribution of the set of parameters (NOT the data) 
f(6) - p.f. or p.d.f. — > corresponds to intuition. 
P has p.f. or p.d.f.; f(x\9) 

Given xi, ...,x n joint p.f. or p.d.f.: f{x\, ...,x n \6) = f(xi\6) x ... x f(x n \6) 

To find the Posterior Distribution - distribution of the parameter given your collected data. 

Use Bayes formula: 




The posterior distribution adjusts your assumption (prior distribution) based upon your sample data. 
Example: B(p), f(x\p) = p x (l - p) 1 "*; 
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/On, ...,x n \ P ) = n P "(i -p) 1 -*' = P ^(i - P ) n -^*< 

Your only possibilities are p = 0.4, p = 0.6, and you make a prior distribution based on 
the probability that the parameter p is equal to each of those values. 
Prior assumption: f(0.4) = 0.7, f(0.6) = 0.3 

You test the data, and find that there are are 9 successes out of 10, p = 0.9 

Based on the data that give p = 0.9, find the probability that the actual p is equal to 0.4 or 0.6. 

You would expect it to shift to be more likely to be the larger value. 

Joint p.f. for each value: 

/(a;i,...,z 1 o|0.4) = 0.4 9 (0.6) 1 



/(a;i ) ...,a;io|0.6) = 0.6 9 (0.4) 1 
Then, find the posterior distributions: 

(0.4 9 (0.6) 1 )(0.7) 



/(0.4|ari, ...,x n ) 
/(0.6|xi, ...,x n ) = 



(0.4 9 (0.6) 1 )(0.7) + (0.6 9 (0Ay)(0.3) 
(Q^OT) 1 )^) 



= 0.08 



(0.4 9 (0.6) 1 )(0.7) + (0.6 9 (0.4) 1 )(0.3) 
Note that it becomes much more likely that p = 0.6 than p = 0.4 

Example: B(p), prior distribution on [0, 1] 

Choose any prior to fit intuition, but simplify by choosing the conjugate prior. 



0.92 



f(p\xi, ...,x n ) 



p^ x -(l-p) n "^f(p) 



Choose f(p) to simplify the integral. Beta distribution works for Bernoulli distributions. 
Prior is therefore: 



T{a + I3) a _ x p_ x 



< l 



Then, choose a and (3 to fit intuition: makes E(X) and Var(X) fit intuition. 

L (l-P) 



f(n\xi X ) - r O + I>»+/3 + "-E^) (o+E»i)-ln _ n ^08+n-E^)-l 



Posterior Distribution = Beta(a + ^Xi, (3 + n — J2 x i) 
The conjugate prior gives the same distribution as the data. 

Example: 



9 out of 10 > 





71 



B(a,(3) such that EX = 0.4, Var(X) = 0.1 

Use knowledge of parameter relations to expectation and variance to solve: 
EX = 0.4 = — ?—,Vai{X) = 0.1 



a + 0' v ' ' (a + /3) 2 (a + /3 + l) 
The posterior distribution is therefore: 

Beta(a + 9,/?+l) 

And the new expected value is shifted: 

EX= a + 9 

a + (3 + 10 

Once this posterior is calculated, choose the parameters by finding the expected value. 
Definition of Bayes Estimator: 

Bayes estimator of unknown parameter 9q is 6{X\, ...,X n ) = expectation of the posterior distribution. 

Example: B(p), prior Beta(a, (3), X\, ...,X n — » posterior Beta(a + ^ Xi, [3 + n — x i) 

ot ~\~ / x ■ (y, ~\~ / x ■ 
Bayes Estimator: = = — = - 

a + 2^ x * + P + n - 1^ x i a + fi + n 

To see the relation to the prior, divide by n: 

a/n + x 
~ a/n + (3/n + 1 

Note that it erases the intuition for large n. 

The Bayes Estimator becomes the average for large n. 

** End of Lecture 23 
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